GH-45523: [R] Implement Utf8View type bindings by thisisnic · Pull Request #49712 · apache/arrow

thisisnic · 2026-04-11T18:15:28Z

Rationale for this change

No bindings for Utf8View type in the R package

What changes are included in this PR?

Implement bindings

Are these changes tested?

Yep

Are there any user-facing changes?

Yep, adding functionality.

AI Usage

Heavily used Codex/Claude here. I'm not confident of every line of code. I read things over, and iterated on it making sure that tests pass and nothing seemed wildly incorrect.

GitHub Issue: [R] An "cannot handle Array of type <utf8_view>" error occurs when reading ipc file include strings output by r-polars #45523

thisisnic · 2026-04-11T18:17:20Z

We should rebase once #49710 is merged as changes from that PR are in this branch as they were needed to make it work.

thisisnic · 2026-05-06T20:49:12Z

+      return this->value_builder_->Append(
+          std::string_view(view_.bytes, static_cast<size_t>(view_.size)));


The old code passed (const char*, int32_t) which matches StringBuilder::Append but not StringViewBuilder::Append (which takes int64_t). Switching to std::string_view works for both builder types.

thisisnic · 2026-05-06T20:52:43Z

A lot of .Rd updates as I used a newer roxygen2 version.

Copilot

Pull request overview

This PR adds R bindings and conversion support for Arrow utf8_view / string_view, addressing IPC/table conversion failures for data containing StringView columns.

Changes:

Adds string_view() R data type binding, export registration, and type tests.
Implements R↔Arrow conversion paths for StringView arrays and dictionary values.
Updates generated documentation and related converter support in C++/Python.

Reviewed changes

Copilot reviewed 13 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
r/R/type.R	Adds `StringView` R6 type, constructor, and canonical type aliases.
r/R/arrowExports.R	Registers R wrapper for `StringView__initialize`.
r/NAMESPACE	Exports `string_view`.
r/src/datatype.cpp	Maps Arrow `STRING_VIEW` to R `StringView` and initializes `utf8_view`.
r/src/r_to_arrow.cpp	Adds R-to-Arrow StringView conversion and dictionary StringView value handling.
r/src/array_to_vector.cpp	Adds Arrow-to-R StringView and wider dictionary index conversion support.
r/src/arrowExports.cpp	Registers native StringView initialization entry point.
cpp/src/arrow/util/converter.h	Enables dictionary converters for `StringViewType`.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Adjusts Python dictionary StringView append call.
r/tests/testthat/test-Array.R	Adds StringView array round-trip tests.
r/tests/testthat/test-Table.R	Adds table/dictionary StringView and wider index tests.
r/tests/testthat/test-data-type.R	Adds StringView data type and code round-trip tests.
r/DESCRIPTION	Updates roxygen metadata.
r/man/data-type.Rd	Adds `string_view()` documentation entry.
r/man/acero.Rd	Regenerated Acero documentation.
r/man/arrow-package.Rd	Regenerated package author documentation.
r/man/csv_convert_options.Rd	Regenerated CSV conversion docs.
r/man/csv_read_options.Rd	Regenerated CSV read docs.
r/man/CsvReadOptions.Rd	Regenerated CSV read options docs.
r/man/enums.Rd	Regenerated enum documentation.
r/man/JsonFileFormat.Rd	Regenerated JSON file format docs.
r/man/reexports.Rd	Regenerated reexports docs.
r/man/vctrs_extension_array.Rd	Regenerated vctrs extension docs.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 13 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

Copilot

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)

r/man/data-type.Rd: Language not supported

thisisnic · 2026-06-18T15:00:31Z

Current CI failure is unrelated; #50219

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

…nary The offset-buffer fast path in Converter_String::Ingest_some_nulls() hardcoded GetValues<int32_t>, which is incorrect for LargeStringArray (int64_t offsets). Use StringArrayType::offset_type so the correct width is resolved at compile time. Also switch the view-type guard from a runtime is_binary_view_like() check to if constexpr, since StringViewArray has no offset_type member and the template would fail to instantiate otherwise. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Exercise out-of-line storage (>12 bytes) in the dictionary string_view test and add NULL + mixed lengths to the binary_view table test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pitrou · 2026-07-29T08:38:57Z

+        }
+      }
+    } else {
+      RETURN_NOT_OK(this->primitive_builder_->Reserve(s.size()));


This should be Reserve(size - offset) just like above, no? Which means it can also be moved out of the if constexpr.

pitrou · 2026-07-29T08:42:51Z

+      p_strings += offset;
+      for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+        SEXP si = *p_strings;


This would be less confusing and error-prone as (IMHO):

Suggested change

p_strings += offset;

for (R_xlen_t i = offset; i < size; i++, ++p_strings) {

SEXP si = *p_strings;

for (R_xlen_t i = offset; i < size; i++) {

SEXP si = p_strings[i];

pitrou · 2026-07-29T08:48:25Z

-    }
-    RETURN_NOT_OK(this->primitive_builder_->ReserveData(total_length));
+    if constexpr (is_string_view_type<T>::value) {
+      RETURN_NOT_OK(this->primitive_builder_->Reserve(size - offset));


Are we sure it's size - offset rather than size? What is the contract for this? cc @paleolimbot @romainfrancois

pitrou · 2026-07-29T08:49:10Z

+      int64_t total_length = 0;
+      for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+        SEXP si = *p_strings;


Same here:

Suggested change

int64_t total_length = 0;

for (R_xlen_t i = offset; i < size; i++, ++p_strings) {

SEXP si = *p_strings;

int64_t total_length = 0;

for (R_xlen_t i = offset; i < size; i++) {

SEXP si = p_strings[i];

pitrou · 2026-07-29T08:49:25Z

+      }
+      RETURN_NOT_OK(this->primitive_builder_->ReserveData(total_length));
+
+      p_strings = reinterpret_cast<const SEXP*>(DATAPTR_RO(s));


No need for this if p_strings isn't mutated above.

pitrou · 2026-07-29T08:49:40Z

+      for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+        SEXP si = *p_strings;


Suggested change

for (R_xlen_t i = offset; i < size; i++, ++p_strings) {

SEXP si = *p_strings;

for (R_xlen_t i = offset; i < size; i++) {

SEXP si = p_strings[i];

github-actions Bot added Component: R Component: C++ awaiting committer review Awaiting committer review labels Apr 11, 2026

thisisnic mentioned this pull request Apr 30, 2026

GH-39603: [R] Error: Cannot convert Dictionary Array of type dictionary<values=large_string, indices=uint32, ordered=0> to R #49710

Merged

thisisnic force-pushed the GH-45523-ipc-polars branch 2 times, most recently from e8768d0 to bb711a7 Compare May 5, 2026 16:41

github-actions Bot added Component: Python awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels May 5, 2026

thisisnic commented May 6, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp

Copilot AI review requested due to automatic review settings May 28, 2026 09:37

thisisnic force-pushed the GH-45523-ipc-polars branch from de59365 to cf9ad10 Compare May 28, 2026 09:37

Copilot started reviewing on behalf of thisisnic May 28, 2026 09:37 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels May 28, 2026

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Outdated

Comment thread r/man/acero.Rd Outdated

Copilot AI review requested due to automatic review settings May 28, 2026 12:11

Copilot started reviewing on behalf of thisisnic May 28, 2026 12:11 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Comment thread r/src/r_to_arrow.cpp Outdated

thisisnic force-pushed the GH-45523-ipc-polars branch from a74c9c1 to 1842e7e Compare June 10, 2026 15:10

Copilot AI review requested due to automatic review settings June 10, 2026 15:18

Copilot started reviewing on behalf of thisisnic June 10, 2026 15:18 View session

Copilot AI reviewed Jun 10, 2026

View reviewed changes

thisisnic marked this pull request as ready for review June 18, 2026 15:01

thisisnic requested a review from jonkeane as a code owner June 18, 2026 15:01

Copilot AI review requested due to automatic review settings June 18, 2026 15:01

thisisnic and others added 14 commits July 28, 2026 19:44

Fix DictionaryBuilder::Append for StringViewType in Python bindings

dfad6fa

Fix docs build issue

3c88f34

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Run make doc

1baff8d

Simplify tests

a017257

Simplify tests further

5494786

appease linter

e5cac7b

Simplify code as per Codex feedback

5c93673

Potential fix for pull request finding

f00d6e0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

d055479

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Update r/tests/testthat/test-Array.R

fafef79

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Implement binary view

5b61c04

Add binaryview

afa85e8

ditch errant test

42ff88b

Revert formatting frama

39c4c92

thisisnic force-pushed the GH-45523-ipc-polars branch from dabaca9 to 39c4c92 Compare July 28, 2026 23:47

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 28, 2026

thisisnic and others added 2 commits July 28, 2026 20:08

Check R_XLEN_T_MAX guard

88d435d

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Jul 29, 2026

Add longer strings to Table string_view/binary_view tests

03be67f

Exercise out-of-line storage (>12 bytes) in the dictionary string_view test and add NULL + mixed lengths to the binary_view table test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 29, 2026

pitrou reviewed Jul 29, 2026

View reviewed changes

		return this->value_builder_->Append(
		std::string_view(view_.bytes, static_cast<size_t>(view_.size)));

		for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
		SEXP si = *p_strings;

Uh oh!

Conversation

thisisnic commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

AI Usage

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

thisisnic May 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

thisisnic commented Jun 18, 2026

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

thisisnic commented Apr 11, 2026 •

edited

Loading